Python multi namespace #5375

msyyc · 2024-12-16T06:50:30Z

Compared to the previous logic that only supported a single namespace, this PR introduces support for multiple namespaces. This primarily requires updates in two key areas:

Serialization approach: We need to account for all namespaces and serialize them individually.
Relative imports: The Python SDK utilizes relative imports internally, and we can no longer make assumptions about the location of models/operations. To determine the correct relative import, we need to know the serialized namespace within the imports function.

Considering these two main changes, here's a detailed summary of the updates in this PR:

Added the emitter flag enable-typespec-namespace, which lets users decide if they want to support native typespec namespace.
Introduced clientNamespace for models/operations/enums/namedUnions/clients: the emitter uses clientNamespace to determine the exact location where the target type should be.
Created get_relative_import_path for CodeModel: all relative import paths should be calculated using this function.
Added a new property client_namespace_types to CodeModel: it counts and stores all namespace types and information about clients/operation_groups/enums/models in each namespace.
Serialization: With the new client_namespace_types property in CodeModel, we can cycle through each namespace and serialize its clients/models/enums/operations using the same logic.
Imports: The old import logic used the relative_path and operation kwargs, which were not set from the top caller but from various functions during the calling process. The signatures were difficult to understand and maintain. I replaces them with the new serialize_namespace and serialize_namespace_type kwargs. Now, any function that needs to calculate relative imports can use these two parameters, which are set from the top caller, making them easier to understand and maintain.

…hon-multi-namespace

azure-sdk · 2024-12-16T06:52:41Z

No changes needing a change description found.

azure-sdk · 2024-12-16T07:03:22Z

You can try these changes here

🛝 Playground	🌐 Website	📚 Next docs

packages/http-client-python/generator/pygen/codegen/models/code_model.py

packages/http-client-python/generator/pygen/codegen/models/client.py

packages/http-client-python/generator/pygen/codegen/models/code_model.py

…hon-multi-namespace

tadelesh · 2024-12-24T06:14:16Z

packages/http-client-python/emitter/src/lib.ts

@@ -43,6 +44,7 @@ const EmitterOptionsSchema: JSONSchemaType<PythonEmitterOptions> = {
    debug: { type: "boolean", nullable: true },
    flavor: { type: "string", nullable: true },
    "examples-dir": { type: "string", nullable: true, format: "absolute-path" },
+    "enable-typespec-namespace": { type: "boolean", nullable: true },


nit: can we add some comment for the new config? i believe we should do such clean up with the pr of consolidating emitter's config name.

Added comment for the new flag. And I talked to JS/.NET devs, then they are fine for current name.

packages/http-client-python/emitter/src/utils.ts

tadelesh · 2024-12-24T06:17:36Z

packages/http-client-python/eng/scripts/ci/regenerate.ts

+  "client/namespace": {
+    "enable-typespec-namespace": "true",
+  },


for unbranded. shall we always enable it by default? i heard java do that by default and for unbranded, it is reasonable.

Here is generated code for e2e demo allenjzhang/typespec-e2e-demo#51 if we enable typespec namespace. We can see there are additional levels of package structure, which is not intuitive. If SDK users want to call some API, they just need to import client then call it:

from todo import TodoAppClient client = TodoAppClient() client.users.get(...)

However, if they need models, they have to make it clear where the model is:

from todo.models import CreateFormResponse from todo.users.models import UserExistsResponse

Compared with before that users can always import any model from todo.models, the usage experience is not better after enable typespec namespace. So I think we shall default --enable-typepsec-namespace as False which I think is better for most developers. And if typespec is really complicated and typepspec author really want the multinamespace in Python SDK, they could add the flag as True in tspconfig.yaml.

i think it is because python does not support initialize the users client directly. to me, users should be a separate client and should be put under the todo.users, then the client, operation, model could be all under todo.users. i prefer to honor the typespec namespace all the time because it could reflect all the spec author's idea. any weird structure should come from a poor spec and author should fix the spec directly.

@iscai-msft / @lmazuel for awareness.

tadelesh · 2024-12-24T06:58:15Z

packages/http-client-python/generator/pygen/codegen/serializers/base_serializer.py


    def init_file_import(self) -> FileImport:
        return FileImport(self.code_model)
+
+    @property
+    def serialize_namespace(self) -> str:


can we add a comment for this? sth like serialize_namespace is the namespace of the serialized file. for example, when we serialize the async operation A with Azure.Test.Foo namespace, then the serialize_namespace should be Azure.Test.Foo.aio._operations.

tadelesh · 2024-12-24T07:01:25Z

packages/http-client-python/generator/pygen/codegen/models/code_model.py

+            self._operations_folder_name[client_namespace] = name
+        return self._operations_folder_name[client_namespace]
+
+    def get_serialize_namespace(


i prefer to move this function to base_serializer.py because it is not related with the code model, but for the code serialization. the result for this function is depend on how we put the generated code. same for other helper like operations_folder_name.

get_serialize_namespace and operations_folder_name use client_namespace_types of code_model and code_model can be accessed in almost all types so it is most convenient way to access them in code_model.

i just want to make sure serializing logic is placed in serializer, code model status and calculation is placed in type model. if they are coupled, then i'm ok with that.

packages/http-client-python/generator/pygen/codegen/serializers/__init__.py

packages/http-client-python/generator/pygen/codegen/serializers/general_serializer.py

tadelesh

the changes lgtm. but i really think we should do a refactor to separate the logic of serializing and type status in some time. current logic of import and other serialization thing has too many coupling things. another thing is we should hold merge till we add complicated test cases in spector.

msyyc added 30 commits November 11, 2024 14:03

update dependency tcgc 0.48.1

feb80b5

add clientNamespace property

60e4c15

init

ac2b225

init2

0a391bd

init3

80ea470

init4

d7f6e87

init5

4146dc4

format

f0d01be

format

3613073

merge

db6402e

review

5ea817c

review

ed61e31

review

682b381

review

32399e8

review

ab818e2

review

f3f0140

review

ea246a9

review

08a68ff

review

a8dc1d4

review

089ebcc

review

59d97c2

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

bbb5884

…hon-multi-namespace

review

9c5f209

review

d8881d1

review

5aeb9cf

review

c8cac2c

review

b0b1b43

review

a21d7ad

review

6c37c0c

review

d0dcc68

msyyc added 9 commits December 11, 2024 13:02

fix

4128461

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

08ca7c7

…hon-multi-namespace

fix typing

6aa4203

fix typing

5700597

review

49cc526

review

5b7fe1a

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

7290a71

…hon-multi-namespace

review

c592198

review

21d8448

msyyc requested review from iscai-msft and tadelesh as code owners December 16, 2024 06:50

microsoft-github-policy-service bot added the emitter:client:python Issue for the Python client emitter: @typespec/http-client-python label Dec 16, 2024

msyyc mentioned this pull request Dec 16, 2024

Auto PR for https://github.com/microsoft/typespec/pull/5375 Azure/autorest.python#2968

Draft

fix crash when no og in client

851fd6c

tadelesh reviewed Dec 23, 2024

View reviewed changes

msyyc added 3 commits December 23, 2024 16:11

review

b493479

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

0449f59

…hon-multi-namespace

review

85435c3

msyyc requested a review from tadelesh December 24, 2024 02:00

msyyc added 3 commits December 24, 2024 11:26

review

c8cd8d1

fix import

a9bdfa7

fix import

06f42b6

tadelesh reviewed Dec 24, 2024

View reviewed changes

msyyc added 3 commits December 25, 2024 15:01

review

34d52e5

review

87d391e

format

8f50702

msyyc requested a review from tadelesh December 25, 2024 07:37

tadelesh approved these changes Dec 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python multi namespace #5375

Python multi namespace #5375

msyyc commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

tadelesh Dec 24, 2024

msyyc Dec 25, 2024

tadelesh Dec 24, 2024

msyyc Dec 25, 2024

tadelesh Dec 25, 2024 •

edited

Loading

msyyc Dec 25, 2024

tadelesh Dec 24, 2024

msyyc Dec 25, 2024

tadelesh Dec 24, 2024

msyyc Dec 25, 2024

tadelesh Dec 26, 2024

tadelesh left a comment

Python multi namespace #5375

Are you sure you want to change the base?

Python multi namespace #5375

Conversation

msyyc commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tadelesh Dec 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tadelesh left a comment

Choose a reason for hiding this comment

tadelesh Dec 25, 2024 •

edited

Loading